{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 环境"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "环境初始化内部状态。在示例场景下，状态就\n",
    "是一个计数器，记录智能体还能和环境交互的步数。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "get_observation()方法能给智能体返回当前环境的观察。它通常\n",
    "被实现为有关环境内部状态的某些函数。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "get_action()方法允许智能体查询自己能执行的动作集。通常，\n",
    "智能体能执行的动作集不会随着时间变化，但是当环境发生变化的时\n",
    "候，某些动作可能会变得无法执行（例如在井字棋中，不是所有的位\n",
    "置能都执行所有动作）。而在我们这极其简单的例子中，智能体只能\n",
    "执行两个动作，它们被编码成了整数0和1。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "is_done():给予智能体片段结束的信号。就像第1章中所述，环境\n",
    "–智能体的交互序列被分成一系列步骤，称为片段。片段可以是有限\n",
    "的，比如国际象棋，也可以是无限的，比如旅行者2号的任务（一个著\n",
    "名的太空探测器，发射于40年前，目前已经探索到太阳系外了）。为\n",
    "了囊括两种场景，环境提供了一种检测片段何时结束的方法，通知智\n",
    "能体它无法再继续交互了。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "action()方法是环境的核心功能。它做两件事——处理智能体的\n",
    "动作以及返回该动作的奖励。在示例中，奖励是随机的，而动作被丢\n",
    "弃了。另外，该方法还会更新已经执行的步数，并拒绝继续执行已结\n",
    "束的片段。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "import random\n",
    "class Environment:\n",
    "    def __init__(self):\n",
    "        self.steps_left=10\n",
    "    def get_observation(self)->list[float]:#返回一个包含浮点数的列表\n",
    "        return [0.0,0.0,0.0]\n",
    "    def get_actions(self)->list[int]:\n",
    "        return [0,1]\n",
    "    def is_done(self)->bool:\n",
    "        return self.steps_left==0\n",
    "\n",
    "    def action(self,action:int)->float:\n",
    "        if self.is_done():\n",
    "            raise Exception(\"GAME IS OVER\")\n",
    "        self.steps_left-=1\n",
    "        return random.random()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 智能体"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "只包含两个方法：\n",
    "构造函数以及在环境中执行一步的方法："
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在**构造函数**中，我们初始化计数器，该计数器用来保存片段中智\n",
    "能体累积的总奖励"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**step函数**接受环境实例作为参数，并允许智能体执行下列操作：<br>\n",
    "·观察环境。<br>\n",
    "·基于观察决定动作。<br>\n",
    "·向环境提交动作。<br>\n",
    "·获取当前步骤的奖励。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "PS:对于我们的例子，智能体比较愚笨，它在决定执行什么动作的时\n",
    "候会忽视得到的观察。取而代之的是，随机选择动作。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Agent():\n",
    "    # 构造函数\n",
    "    def __init__(self):\n",
    "        self.total_reward=0.0\n",
    "    # 在环境中执行一步的方法\n",
    "    \"\"\"\n",
    "    env: Environment 是类型注解，表示这个方法接受一个 Environment 类的实例作为参数。\n",
    "    \"\"\"\n",
    "    def step(self,env:Environment):\n",
    "        current_obs=env.get_observation()\n",
    "        actions=env.get_actions()#获取可用动作\n",
    "        reward=env.action(random.choice(actions))#执行动作\n",
    "        self.total_reward+=reward#更新智能体的总奖励。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "分数： 5.215687561657628\n"
     ]
    }
   ],
   "source": [
    "if __name__=='__main__':\n",
    "    env=Environment()\n",
    "    agent=Agent()\n",
    "while not env.is_done():\n",
    "    agent.step(env)\n",
    "print(\"分数：\",agent.total_reward)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
